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Improving Student Achievement 
and Teacher Effectiveness Through 
Scientifically Based Practices 

INTRODUCTION 

Viewpoints is a multimedia package containing two audio CDs and a short, 
informative booklet. This volume of Viewpoints focuses on using scientifically 
based practices to improve student achievement and teacher effectiveness. 

The audio CDs provide the voices, or viewpoints, of various leaders from the 
education field who have worked closely with scientifically based research ini- 
tiatives. Their voices represent perspectives on the elements of scientifically 
based research in education and provide a general overview of the movement. 

This booklet presents information about the recent increased emphasis on 
using scientifically based research to improve student achievement and 
teacher effectiveness. It explains new vocabulary, discusses research chal- 
lenges, and describes how scientifically based research can be turned into 
effective practice. The booklet also provides an overview of the What Works 
Clearinghouse and offers resources and tools for using data to make decisions 
in schools. 

THE ISSUE 

Teachers and administrators alike are challenged by the No Child Left 
Behind legislation to incorporate scientifically based research into their deci- 
sion making for programs and practices that will improve student achieve- 
ment. Education researchers also are challenged to produce studies that are 
faithful to scientifically based principles. Now more than ever, practitioners 
and researchers need to link their efforts to address student learning. Finding 
new opportunities for educators, policymakers, and researchers to work 
together on behalf of schools affected by the new legislation is a challenge, 
and it instills hope that scientifically based research will provide better direc- 
tion and evidence that student achievement can improve. 



THE BOOKLET: A GUIDE TO CONTENTS 

The essay “The Challenge and Hope of Scientifically Based Research” serves 
as a companion to the CDs. This essay outlines the elements of scientifically 
based research and some of the challenges educators face in its implemen- 
tation. There are guides to understanding the levels of research and 
resources to investigate the topic further. You may find it helpful to read the 
booklet as an introduction to the topic before listening to the interviews pre- 
sented on the CDs. 
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The Challenge and Hope of 
Scientifically Based Research 

By Margaret A. Trybus, Ed.D. • 

Assistant Superintendent, Curriculum and Instruction, • 
Thornton Fractional Township High School District 215* 

Teachers and administrators alike are challenged by the No Child Left 
Behind Act of 2001 to incorporate scientifically based research (SBR) into 
their decision making for programs and practices that will improve student 
achievement. Educational researchers also are challenged to produce stud- 
ies that are faithful to scientifically based principles. Now more than ever 
before, practitioners and researchers need to link their efforts to address 
student learning as a result of the No Child Left Behind legislation. The chal- 
lenge is to base practice on rigorous evidence that specific programs will 
work to guide teaching and learning and, at the same time, to understand 
the reality that this type of research is not readily available or understood by 
most administrators and classroom teachers. Finding new opportunities for 
educators, policymakers, and researchers to work together on behalf of 
schools affected by the new legislation is a challenge, and it instills hope that 
SBR will provide better direction and evidence that student achievement 
can improve. 

This essay outlines the place of SBR in the No Child Left Behind Act of 
2001, provides details about the mandate for SBR in the Comprehensive 
School Reform legislation, and explains the rationale and challenges for 
using SBR in making education decisions. It also provides important defini- 
tions and outlines tools for translating research into practice. 

Prioritizing SBR 

The No Child Left Behind Act redefines the federal role in K-12 education by 
stipulating that federally funded programs and practices must be grounded in 
SBR. Funding is determined in part by whether programs and practices have 
a basis in “scientific research" (Beghetto, 2003). The No Child Left Behind 
(NCLB) Act mentions the term scientifically based research 111 times, and 
SBR is mentioned extensively in Title I: “to promote school wide reform and 
ensure the access of children to effective, scientifically based instructional 
strategies and challenging academic content” (NCLB Act, 2002, Title I, 
Section 1001 [9]). Title I, the largest federally funded program for education- 
ally disadvantaged children, requires both state and local education agencies 



to use SBR for targeted-assistance schools to strengthen the core academ- 
ic program. It also calls for school improvement efforts that seek to “identi- 
fy and implement professional development, instructional strategies, and 
methods of instruction that are based on scientifically based research and 
that have proven effective in addressing the specific instructional issues that 
caused the school to be identified for school improvement” (NCLB Act, 
2002, Title I, Part A, Section 1116 [4] [B] [ii]). 

All technical assistance — whether provided by a state agency, higher edu- 
cation, or a state-approved professional development provider — also must 
comply with the SBR mandate by guaranteeing that new curriculum, 
instructional strategies, or specific initiatives that promise to improve the 
learning of low-achieving students meet SBR requirements. Subparts of Title I 
also cite SBR under Reading First, Early Reading First, Even Start Literacy 
Program, Improving Literacy through School Libraries, and Comprehensive 
School Reform (NCLB Act, 2002). Whether funds are used for procuring 
instructional materials such as software or reading programs, or funds are 
used for establishing partnerships between professional development 
providers and local school agencies, all efforts need to be screened through 
the SBR mandate. 

SBR in Comprehensive School Reform 

The SBR legislation has posed a challenge to schools seeking to implement 
reform strategies through the Comprehensive School Reform (CSR) 
Program. With the availability of additional funding, schools can make a 
commitment to work with an outside provider who must demonstrate that a 
particular reform program has proven SBR results and is replicable to the 
school using CSR. This initiative has far-reaching implications since it 
addresses every aspect of the school: all grades and key subjects (primari- 
ly English and mathematics curricula and instructional practices), school 
management, parental involvement, community involvement, and school 
organization (Herman, 2002). 

CSR is usually initiated when individual school improvement efforts are not 
successful and assessment of data indicates that students are not meeting 
standards. Making Good Choices: A Guide for Schools and Districts (Hassel, 
1998) offers a process for selecting a CSR model that requires identifying 
two or three models to find the best match between the model provider and 
the local school needs. Since 2001 and the reauthorization of the 
Elementary and Secondary Education Act, the standard for being an 
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The 1 1 Components of a Comprehensive School 
Reform Program 



According to the NCLB Act (2002), a Comprehensive School Reform school must 
implement a program that: 



(1) employs proven strategies and proven methods for student learning, teaching, and 
school management that are based on scientifically based research and effective prac- 
tices and have been replicated successfully in schools; 

(2) integrates a comprehensive design for effective school functioning, including 
instruction, assessment, classroom management, professional development, parental 
involvement, and school management, that aligns the school’s curriculum, technology, 
and professional development into a comprehensive school reform plan for school- 
wide change designed to enable all students to meet challenging State content and stu- 
dent academic achievement standards and addresses needs identified through a 
school needs assessment; 

(3) provides high quality and continuous teacher and staff professional development; 

(4) includes measurable goals for student academic achievement and benchmarks for 
meeting such goals; 

(5) is supported by teachers, principals, administrators, school personnel staff, and 
other professional staff; 

(6) provides support for teachers, principals, administrators, and other school staff; 

(7) provides for the meaningful involvement of parents and the local community in 
planning, implementing, and evaluating school improvement activities consistent with 
section 1118; 

(8) uses high quality external technical support and assistance from an entity that has 
experience and expertise in schoolwide reform and improvement, which may include 
an institution of higher education; 

(9) includes a plan for the annual evaluation of the implementation of school reforms 
and the student results achieved; 

(10) identifies other resources, including Federal, State, local, and private resources, 
that shall be used to coordinate services that will support and sustain the comprehen- 
sive school reform effort; and 

( 1 1) (A) has been found, through scientifically based research to significantly improve 
the academic achievement of students participating in such program as compared to 
students in schools who have not participated in such program; or 

(B) has been found to have strong evidence that such program will significantly 
improve the academic achievement of participating children. (Title 1, Part F, Section 
1606 [a]) 



approved CSR model, however, has been changed from using “innovative 
strategies and proven methods for student learning, teaching, and school 
management based on reliable research and effective practices” to a call for 
comprehensive reform programs that “employ proven strategies and proven 
methods for student learning, teaching, and school management that are 
based on scientifically based research and effective practices and have 
been replicated successfully in schools” (Comprehensive School Reform 
[CSR] Program Office, & Office of Elementary and Secondary Education, 

2002, p. 2). 

Of the 1 1 components schools must implement to receive CSR funding, two 
specifically require a high standard of SBR. One component that requires a 
high standard is the selection of a model that uses proven strategies and 
methods for learning, teaching, and school management based on SBR and 
effective practices, and that has been used successfully in multiple schools. 
Another component requires that an approved CSR model has “been found, 
through scientifically based research, to significantly improve the academic 
achievement of students ... or has been found to have strong evidence that 
[it] will” (CSR Program Office & Office of Elementary and Secondary 
Education, 2002, p. 2). For the other nine components (see page 5), the 
U.S. Department of Education recognizes “school leaders will need to rely 
on the best available empirical evidence and some degree of professional 
judgment in creating their programs” (CSR Program Office & Office of 
Elementary and Secondary Education, 2002, p. 3). 

In 2000, the American Institutes for Research (AIR, 2000) published the 
Educators' Guide to School Reform, reviewing the research on 24 of the 
most promising CSR models in the country. Even though the research was 
limited, each model was analyzed against criteria including the research 
design, controls, validity, sample size, measures, outcomes, and duration of 
the study (Herman, 2002). The American Association of School 
Administrators (AASA, 2003) has provided an online summary of the guide 
for the 24 “whole-school,” “comprehensive,” or “schoolwide” approaches 
reviewed by AIR. This guide (at www.aasa.org/issues_and_insights/ 
district_organization/Reform/overview.htm) provides a review of the relative 
strengths of each approach in three areas: 

■ Evidence of positive effects on student achievement. 

■ Support provided to schools as they adopt the approaches. 

■ First-year adoption costs. (AASA, 2003) 
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This guide will help decision makers determine areas of strength and weak- 
ness, as well as the amount of evidence available to assess the effects of a 
model on student achievement. 

The Logic of SBR 

Traditionally, relying on anecdotal evidence and professional judgment has 
been the approach many schools have taken with respect to school 
improvement initiatives, especially those schools not engaged in CSR. Since 
the passage of the NCLB Act, schools must be more diligent in their use of 
research and alter their decision-making practices accordingly. According to 
Assistant Secretary Grover Whitehurst, we have to create a culture and prac- 
tice that demands more from our educational system by using evidence- 
based practices in order to transform education in the same order of mag- 
nitude as in medicine and agriculture (U.S. Department of Education, 
2002 ). 

The sense that education is moving to evidence-based practices poses 
some concerns for school leaders who are skeptical that they will have 
access to appropriate research that is replicable and solid (Beghetto, 2003). 
Some fear that the “heart and soul’’ of education may be lost, and along with 
it the interpersonal relationships between administrators, teachers, and stu- 
dents. Not so according to Valerie Reyna (2002), former deputy of the Office 
of Educational Research and Improvement (now the Institute for Education 
Sciences), who emphatically believes that there is no dichotomy between 
science and values or science and emotion: 

Evidence does not determine our decision solely. It is not just the facts. 
It’s the facts plus values. But without the facts, we might make the 
wrong decision, even based on our values. Because we don’t know 
what’s true and what’s not true. The facts, the evidence is necessary to 
make decisions that affect students’ lives, but it’s not sufficient. But it is 
necessary. That is what we’re promulgating, that, at least, it be part of 
the discussion so that we can base practice on it. So, we’re talking 
about science with a human face, and that’s a person, (p. 10) 

As we bring scientific reasoning into educational decision making, it is crit- 
ical that we balance hard evidence with human judgment (Feuer, 2002). 
Together policymakers and practicing educators will have to find ways to 
address the balance between research and practical wisdom (Shavelson & 
Towne, 2002). To make this happen will require building partnerships 
between researchers and practitioners to address both improving practice 
through research and improving research through a better understanding of 
what happens on a daily basis in schools (Towne, 2002). 



Defining SBR 

According to the NCLB Act (2002), the term scientifically based research-. 

(A) means research that involves the application of rigorous, systematic, and 
objective procedures to obtain reliable and valid knowledge relevant to edu- 
cation activities and programs and 

(B) includes research that - 

(i) employs systematic, empirical methods that draw on observation or 
experiment; 

(ii) involves rigorous data analyses that are adequate to test the stated 
hypotheses and justify the general conclusions drawn; 

(iii) relies on measurements or observational methods that provide reli- 
able and valid data across evaluators and observers, across multiple 
measurements and observations, and across studies by the same or dif- 
ferent investigators; 

(iv) is evaluated using experimental or quasi-experimental designs in 

which individuals, entities, programs, or activities are assigned to dif- 
ferent conditions and with appropriate controls to evaluate the effects of 
the condition of interest, with a preference for random-assignment 
experiments, or other designs to the extent that those designs contain 8 

within-condition or across-condition controls; 

(v) ensures that experimental studies are presented in sufficient detail 
and clarity to allow for replication or, at a minimum, offer the opportu- 
nity to build systematically on their findings; and 

(vi) has been accepted by a peer-reviewed journal or approved by a 
panel of independent experts through a comparably rigorous, objective, 
and scientific review. (Title IX, Part A, Section 9101 [37]) 

This definition is intended to encourage researchers to provide better and 
more useful evidence of what works and to challenge practitioners to make 
good decisions based on evidence (Feuer, 2002). The difficulty is that few 
studies of education programs meet this definition in its entirety. 

In 2002, AIR introduced two standards against which education research 
can be judged. The gold standard is research that meets all the require- 
ments of SBR; the silver standard is research that meets the requirements 
but does not employ random sampling (AIR, 2002). The institute's work, 
which was prepared for the U.S. Department of Education, also includes 
some guidelines that can be used by school staff and others to review edu- 
cation research. These guidelines include: 
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■ The theoretical base of the program or practice, explaining specific 
goals followed by implementation activities. 

■ The evidence of effects, stating how the practice has demonstrated 
improved student learning. 

■ Implementation and replicability, explaining the degree to which the 
program has been successfully implemented in diverse settings. 
(AIR, 2002) 

The following graphic, developed by the Iowa Department of Education, 
illustrates the potential range of methodologies used in education 
research. 

FIGURE 1 . Iowa Content Network: Continuum for Reviewing Research 




Gold Standard 

NCLB criteria; most rigorous 
research designs; research-proven 
programs/strategies. 

Strong Evidence 

Research based programs and strategies; 
strong evidence but lacking one/ 
two design elements present 
gold standard." 

Promising 

Research related programs with 
one or more proven strategies 
but untested as a whole; 
need systematic study. 

Marginal 

Programs with one or two 
promising practices but little 
supporting empirical evidence. 

No Empirical Evidence 

Untested programs or 
theories; no empirical 
evidence reported. 

Reports are anecdotal, 
testimonial, etc. 



From the Iowa Professional Development Model: Selecting Content 
(www. state . ia . us/ed u cate/ecese/tqt/tc/p rod ev/d ef i n iti on s . htm I ) . 

Reprinted with permission of the Iowa Department of Education. 



According to Slavin (2003), scientific research traditionally has played a 
relatively minor role in education reform since many innovative practices 
and programs are untested. When reform efforts fail, educators and poli- 
cymakers move to implement a different set of innovations that also have 
untested claims, instead of adopting well-researched programs and prac- 
tices that have been proven to work. Shifting to a new paradigm will mean 
changing practice to look more deeply to research-based programs rather 
than following a new trend. 

Unlike most other fields of scientific inquiry, education places extraor- 
dinary emphasis on the new and the novel. Believing that the most 
recent theory — at whatever level of research — is also the most impor- 
tant, education leaders may lose sight of the value of seminal research 
and proven practices. (Grossen, 1996, p. 22) 

Both Congress and the U.S. Department of Education are hopeful that with 
the introduction of new research standards and federal mandates, “evi- 
dence-based reform” will become the norm and set an expectation for using 
rigorous, experimental research to justify programs and practices. 

Broadening SBR Within NCLB 

In addition to Comprehensive School Reform and Title I legislation, SBR also 
is cited in Title II, Preparing, Training and Recruiting High Quality Teachers 
and Principals; Title III, Language Instruction for Limited-English-Proficient 
and Immigrant Students; Title IV, 21st Century Schools; Title V, Promoting 
Informed Parental Choice and Innovative Programs; Title VI, Flexibility and 
Accountability; Title VII, Indian, Native Hawaiian, and Alaska Native 
Education; and Title IX, General Provisions (NCLB Act, 2002). 

Each applicant for funding must demonstrate efforts to address SBR by 
describing how each activity will be based on a review of SBR and will be 
tied to evidence-based results. This has far-reaching implications for pro- 
fessional development in all content areas, teacher preparation programs in 
higher education, English language acquisition programs, safe and drug- 
free programs, parent involvement, mentoring programs, and all programs 
designed to address state and local student academic achievement stan- 
dards (NCLB Act, 2002). 

Because of the extent of this legislation, all schools will be affected — not just 
those that have been in CSR programs or identified as in need of improve- 
ment. Using SBR will require establishing a culture of inquiry regarding how 
decisions are made to improve student learning. Leadership will matter at both 
the system and school level, and must include teachers with high-quality 
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professional learning to improve practice (Sparks, 2003). Infusing SBR into 
school culture will require enhanced professional learning to increase under- 
standing of the meaning and usefulness inherent in compelling research to 
drive practice. 

Learning a New Vocabulary 

In order to become a critical consumer of research, one must understand 
the language and terminology of research. In addition to understanding the 
definition of SBR and various standards for research, practitioners need a 
basic vocabulary with which they can interpret and communicate research 
studies to one another, to parents, and even to students when appropriate. 
The following definitions are a few suggestions. 

Evidence-based education: “The integration of professional wisdom 
with the best available empirical evidence in making decisions about how 
to deliver instruction” (Whitehurst, 2002). 

Professional wisdom: “The judgment that individuals acquire through 
experience. Increased professional wisdom is reflected in numerous 
ways, including the effective identification and incorporation of local cir- 
cumstances into instruction” (Whitehurst, 2002). 

Control group: A group of individuals whose characteristics are similar to 
the experimental group except that they do not receive any of the program 
services or products being evaluated. Slavin (2003) suggests that, in a 
good study, several schools using a given program are compared with 
schools who are not using the program but meet the same demographic 
criteria. (Having at least five schools in each group is desirable.) 

Empirical research/evidence: Research conducted for the purpose of col- 
lecting measurable data in terms of attitudes, behavior, or performance. 
Empirical research is designed to generate projectable, numerical data 
on a topic. 

Randomized experiment: The most convincing form of a control group 
comparison in which students, teachers, or schools are assigned by 
chance to a group. Such comparisons are very rare in education, but 
very influential (Slavin, 2003). Some educational researchers contend 
that even though this method is used in the medical field, it is difficult to 
conduct in educational contexts and may be potentially harmful to chil- 
dren (Mid-continent Research for Education and Learning, 2002; 
Shavelson & Towne, 2002). This is part of the criteria to reach the gold 
standard of SBR. 



Statistical significance: The difference between the achievement of 
students in the experimental and the control group. “A usual criteri- 
on is p<0.05, which means that the probability is less than 5 percent 
that an observed difference might have happened by chance" 
(Slavin, 2003, p. 14). 

Effect size: Studies should be reviewed to determine the number of 
schools involved in the research and whether the effect size is higher than 
+0.20 (Slavin, 2003; Marzano, 2003). The more schools involved in a 
study done by more than one researcher, the more confident you can be 
that the program’s results are valid. 

As one becomes familiar with the terminology and the concepts undergirding 
SBR, balancing the use of professional wisdom and empirical evidence will 
determine the extent to which evidence-based programs can be useful. Figure 
2 illustrates the relationships between individual experience, consensus, sci- 
entifically based research, and empirical evidence in executing evidence- 
based education programs. 

FIGURE 2. Evidence-Based Education 




12 



Source: Whitehurst, G. J. (2002). Evidence-based education (slide presentation). 

A basic understanding of research terminology and methods will help prac- 
titioners develop skills to make better judgments regarding educational 
research. It will also help to develop an understanding of the work being 
done by professional panels, such as the What Works Clearinghouse, that 
are identifying appropriate research for replication. There is no doubt, how- 
ever, that the movement to create a more evidence-based environment will 
grow as researchers work to meet the needs of thousands of schools 
throughout our country who are challenged to improve learning for all 
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students. Educators will need to be critical in judging whether the claims 
made by research make sense based on research design and methodology 
coupled with practitioners’ experience in the field (Carnahan & Fitzpatrick, 
2003). The more practitioners can do to increase their understanding of the 
findings, the more the intent of the law can be met and improvements in 
education addressed. 

Research Challenges 

How will the work of educational researchers adjust to meet the “evidence- 
based standard” and at the same time recognize the need to increase the 
availability of research for practitioners? How will education research be 
guided by scientific methodologies, as in the field of medicine, yet not limit 
the advancement of the education research field? One issue is the extent to 
which education research can use a medical research model that imple- 
ments a “randomized-control” approach. Using this approach, researchers 
randomly assign students to treatment and control groups in order to deter- 
mine the effectiveness of what is being studied (Reyna, 2002; Lewis, 2003). 
Some claim this approach is too narrow a focus for education research and 
ultimately will limit growth in the field (Laitsch, 2003). There is also a 
concern that qualitative studies will be dismissed in favor of more sophisti- 
cated statistical analyses of measurements and deemed unworthy of serious 
consideration because they do not meet the SBR requirements (Davis, 
2002; Flinders, 2003). 

Over the last 20 years, researchers such as Eisner (1997) have worked to 
justify qualitative methods that have broadened the field of research by rec- 
ognizing the need to include social sciences as well as arts and humanities 
as sources of research methodology. Some fear that the move to formalize 
quantitative research as the gold standard for education research may 
diminish the pursuit of qualitative forms of data. There are also issues 
around the difficulty of using randomized studies in public schools. 
Randomized studies often are more controversial since parents may be 
uncomfortable with researchers identifying students who will and who will 
not receive a particular treatment (Flinders, 2003; Shavelson & Towne, 
2002). Some compromise may be necessary to consider quasi-experimen- 
tal studies, that is, “studies that do not truly randomize sampling or assign- 
ments, but that try to approximate these procedures” (Flinders, 2003, p. 
384). According to Valerie Reyna, quasi-experimental studies provide a 
lower level of evidence, but are “second best” to randomized experimental 
trials, which allow for more stringent statistical control of the variables, “it’s 
not as good, but at least you know that something is probably true based on 
some form of evidence” (Reyna, 2002). 



Founded in 1916, the American Educational Research Association (AERA, 
n.d.), “strives to improve the educational process by encouraging scholarly 
inquiry related to education.” In response to the mandate for SBR, AERA 
(2003) adopted a resolution on January 26, 2003, which reasserted: 

that there are multiple components of quality research, including well 
specified theory, sound problem formulation, reliance on appropriate 
research designs and methods, and integrity in the conduct of research 
and the communication of research findings. A fundamental premise of 
scientific inquiry is that research questions should guide the selection 
of inquiry methods. ... However, the Council of the Association express- 
es dismay that the Department of Education through its public state- 
ments and programs of funding is devoting singular attention to this one 
tool of science, jeopardizing a broader range of problems best 
addressed through other scientific methods. The Council urges the 
Department of Education to expand its current conception of scientifi- 
cally-based research (p. 1). 

The hope that sound, rigorous educational research will reach practitioners 
raises the issue of “usability.” Researchers and practitioners alike agree 
that, “to be effective, education research needs to be both credible and 
usable” (Viadero, 2003). With the focus on scientific experimentation, there 
is a concern that federal funding will not be available to support other kinds 14 

of research that practitioners also may find useful. Research studies may 
become “too academic,” not practical enough for replication and not in a 
language accessible to practitioners. Educators may “view those writings as 
inaccessible, arcane, and irrelevant to their everyday jobs” (Viadero, 2003). 

On the other hand, education research has been criticized for securing 
scant evidence of affecting practice and improving student achievement, 
and skepticism regarding the use of federal funds to support research and 
development efforts has existed for over a decade (Kaestle, 1993). 

Access to Evidence -Based Research 

Realizing both researchers' challenge to produce SBR, and practitioners’ 
challenge to find and utilize SBR, the U.S. Department of Education, 
through the Institute of Education Sciences, has established the What Works 
Clearinghouse (WWC). It will become a resource to assist in educational 
decision making regarding scientifically based programs and practices 
(Beghetto, 2003). The WWC (available online at www.w-w-c.org) was creat- 
ed in August 2002 as a national project to provide the information decision 
makers need to make choices based on high-quality scientific research 
(WWC, 2002a). This will be done through Web-based databases that will 
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Practically Speaking: How Might Practitioners Put Scientifically 
Based Research to Work? 



After a school learns that it is on the academic early 
warning list because some subgroups have not met stan- 
dards on die state assessment for mathematics, a school 
improvement committee is formed to write a school 
improvement plan (SIP) that meets state criteria. (See 
the SIP from the Illinois State Board of Education at 
www.isbe.net/sos/improvement/SIPRHBRIC.pdf for an 
example of this type of plan.) 

The committee is especially concerned with creating 
an “action plan” that designates activities “supported 
by scientifically based research with a theoretical 
base,” as mandated in the state criteria. The state 
rubric specifically asks how the activities cited in the 
plan are supported by SBR, and what types of meas- 
ures will be used to determine if the activities meet 
the needs of the low-achieving students. 

The following steps are taken to address this mandate: 

1 . Analyzing school data. The school improvement 
committee does an extensive analysis of school 
information including attendance, truancy mobili- 
ty, expulsion, and retention rates. Committee mem- 
bers also analyze data on the demographics of the 
school population, breaking it into economically 

15 disadvantaged, limited-English-proficient, white, 
African-American, Hispanic, Native American, and 
Asia/Pacific Islander students. 

2. Generating a hypothesis. The data analysis 
leads committee members to generate a series of 
hypotheses to provide possible explanations for 
why some students are not meeting state standards. 
They discuss the primary causal factors that con- 
tributed to low achievement, and select the factors 
within the school’s capacity to change or control. 
The four factors they choose to address are (a) 
homework completion, (b) mathematics problem 
solving, (c) algebra skills, and (d) parental 
involvement. 

3. Creating an action plan. The school improve- 
ment committee begins its search of scientifically 
or evidence-based research, looking for instruc- 
tional activities that would fit the factors identified. 
Committee members search the What Works 
Clearinghouse (www.w-w-c.org) to find relevant 
research, and read articles from the National 
Council of Teachers of Mathematics 
(www.nctm.org) in order to understand the theo- 
retical base of best practices in mathematics. 



4. Evaluating research. The committee reviews the 
research, investigating the plausibility of replicat- 
ing it in the school improvement plan. To evaluate 
the research, the committee frames its questions 
around major themes: 

■ Theoretical framework. What was the theoreti- 
cal basis of the program being considered? Is it 
related to learning theory and best practices in 
the field of mathematics? 

■ Research design. Did the study have a control 
and an experimental group that were randomly 
assigned? If not, what was the research 
method? What was the sample size? What was 
the study’s hypothesis? What were the 
researchers trying to test or prove? 

■ Research methodology. What was the treatment 
for the experimental group? What activities 
were implemented? How many schools were in 
the study? In what settings did the research 
take place? Was the context similar to our 
school demographics? 

■ Implementation and replication. Is the treat- 
ment program understandable so that it can be 
replicated in our school? What kinds of 
resources were used in the study, and do we 
have the capacity to provide them? 

■ Evidence of residts. What assessment data 
were used to measure the treatment? If 
achievement testing, how comparable is this to 
our assessment system? Has the hypothesis 
been adequately tested in order to justify the 
conclusions? What statistical measurements 
were used, and do we have the capability to 
replicate them? 

■ Approved research. To what extent has the 
research been accepted by a peer-reviewed jour- 
nal or approved by a panel of independent 
experts? Is the research considered rigorous, 
objective, and scientific to meet the SBR criteria? 

After completing these processes, the school leaders 
make a commitment to translate specific, applicable 
research into practices in their own setting. With these 
steps, the school has begun to incorporate research 
into its culture and to increase the likelihood of 
strengthening teaching and improving learning. 



contain specific reviews of “potentially replicable interventions (programs, 
products, and practices)” that have been identified by synthesizing the sci- 
entific evidence that indicates effectiveness in improving student outcomes 
(WWC, 2002a). Other components will include an educational approaches 
and policies registry for evidence-based research reviews, a test instruments 
registry, and an evaluator registry that will identify individuals and organiza- 
tions that are willing to conduct quality evaluation of education interventions 
(What Works Clearinghouse, 2002a; Rentner, Chudowsky, Fagan, Gayler, 

Hamilton, & Kober, 2003). The WWC has an ambitious agenda, which 
includes the following: 

■ Establishing standards for research aiming to draw causal inferences 
and to establish the validity of tests and other assessments. 

■ Creating an independent group of technical advisors. 

■ Selecting topics to focus research. 

■ Conducting syntheses of research studies. (WWC, 2002b) 

The standards, which were approved in June 2003, will be used to guide 
the WWC Evidence Reports, which will be prepared by teams of analysts 
who will create systematic reviews of evidence. “Each WWC Evidence 
Report will examine the effects of programs, practices, products, and poli- 
cies that are designed to improve student outcomes within a topic area” 16 

(WWC, 2002c). The first-year topic areas will include: 

■ Interventions for beginning reading 

■ Curriculum-based interventions for increasing K-12 mathematics 
achievement 

■ Programs for preventing high school dropouts 

■ Programs for increasing adult literacy 

■ Peer-assisted learning in elementary schools: reading, mathematics, 
and science gains 

■ Interventions to reduce delinquent, disorderly, and violent behavior in 
middle and high schools 

■ Interventions for elementary school English language learners: 
increasing English language acquisition and academic achievement 

Another feature is the What Works Clearinghouse’s Cumulative Research 
Evidence Assessment Device (CREAD), which is meant “to provide an 
expression of the confidence with which a conclusion can be drawn about 
the existence of causal effects of an intervention based on an entire body of 
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accumulated evidence” (WWC, 2003, p. 1). Using the evidence-base 
approach will allow for the review of multiple studies that address the same 
topic, thus increasing the confidence that an intervention had a large, small, 
or no effect. These are important issues for practitioners when determining 
the degree to which the program or practice is appropriate for a particular 
school and population of students. 

Practitioners are seeking assistance with implementing quality research. 
The Association for Supervision and Curriculum Development (ASCD) 
recently began publishing Research Brief. This Web-based publication 
(www.ascd.org/direct/researchbrief.html) will help educators and policy- 
makers translate high-quality research into usable decision-making tools. In 
addition, to support the use of evidence-based practices, ASCD has recent- 
ly published two books: (1) What Works in Schools: Translating Research 
into Action and (2) Classroom 
Instruction that Works: Research- 
Based Strategies for Increasing 
Student Achievement. in 
Classroom Instruction that Works, 
researchers at Mid-continental 
Research for Education and 
Learning used a meta-analysis 
technique to analyze selected 
research studies to determine the 
effect of a particular intervention 
(Marzano, Pickering, & Pollock, 

2001). With this methodology, the 
effect size — which was also 
equated to percentiles — deter- 
mined the increase or decrease 
in achievement of the experimental group of students who were involved 
with a specific instructional technique. By identifying specific instructional 
strategies that have a high effect size, Marzano, Pickering, and Pollock were 
able to establish categories of strategies with which practitioners can identi- 
fy. Even though the authors admit that not all instructional strategies will 
work well in all situations, they have presented useful tools for practitioners 
to consider when implementing instructional strategies intended to improve 
student achievement. 

In What Works in Schools: Translating Research into Action, Marzano 
(2003) contends that 35 years of research should not be overlooked in 



Categories of Effective 
Instructional Strategies 

■ Identifying similarities and differences 

■ Summarizing and note taking 

■ Reinforcing effort and providing recognition 

■ Homework and practice 

■ Nonlinguistic representations 

■ Cooperative learning 

■ Setting objectives and providing feedback 

■ Generating and testing hypotheses 

■ Questions, cues, and advance organizers 
(Marzano, Pickering, & Pollock, 2001) 



determining guidance on effective schooling. His work synthesizes this 
research and claims that evidence exists to support three general factors 
that influence student academic achievement: school-level factors, teacher- 
level factors, and student-level factors (Marzano, 2003, p. 10). Marzano 
identifies a process for implementation of his school reform model, which 
takes these factors into consideration. He defines steps for schools to iden- 
tify the elements of the model that are applicable and identifies actions to 
take and steps to implement those actions and to determine the effects of 
the actions. What Works in Schools is one way practitioners can think about 
applying research-based factors to the process of school reform. 

Moving Forward 

Is it possible to improve the education of children by following the guidance 
of scientific methodology? Can we help students read better, do mathemat- 
ics better, and achieve better by focusing efforts on research that has been 
proven effective? Will research become the basis upon which practitioners 
make decisions so that the most effective practices will be used to help all 
children learn? 

The challenge and hope of accepting, understanding, and using SBR is now 
part of educational reform and accountability measures to see that no child 
will be left behind. It will take a concerted effort on the part of all stake- 
holders — administrators, teachers, parents, researchers, business leaders, 
and policymakers — to move this vision into action. It will take both scientif- 
ic and professional wisdom to create a knowledge base, as well as a belief 
system, that this educational mandate has the power to transform teaching 
and learning. It will require building a data-driven system in which teachers 
look for results and continually challenge themselves to make decisions for 
improvement based on the integration of meaningful and relevant data and 
research. It will require educators to let go of past practices that are no 
longer justifiable because they are not achieving identified goals. It will 
require risk taking to understand and apply research methods in classrooms 
so that practitioners will not only be guided by evidence, but also informed 
of their own results. It will require a new vision of staff development that 
seeks to empower educators to be the agents of change and to try new ideas 
that offer to improve student learning. It will challenge educators to defend 
school improvement decisions so that instructional practices are grounded 
in results. The impetus to move forward must come from the willingness to 
use SBR and apply it to classroom settings. After all, children deserve no 
less than access to that which is proven effective and that which offers hope 
that improvements in achievement can be made. 
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Audio CDs: A Guide to Contents 

The CDs provide various perspectives on the elements of scientifically based research. 

CD 1 - INTERVIEWS (in order of appearance) 

1. Howard Bloom, chief scientist at MDRC and pioneer in the application of 
controlled experimental design to social issues. 

2. Robert Slavin, founder of Success for All and proponent of scientific stud- 
ies of program effectiveness. 

3. Lisa Towne, study director at the National Research Council and coauthor 
of Scientific Research in Education. 

4. Phoebe Cottingham, commissioner of educational evaluation and regional 
assistance, the Institute of Education Sciences, U.S. Department of 
Education. 
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5. Rebecca Herman, principle research scientist at the American Institutes ^ 
for Research and project director of the What Works Clearinghouse. 
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CD 2 - INTERVIEWS (in order of appearance) 2. 

1 . Ellen Condliffe Lagemann, dean of the Harvard Graduate School of * 

Education. 22 

2. Peter Robertson, chief information officer for Cleveland Public Schools. 

3. Todd DeMitchell, professor of education law and chair of the Department 
of Education at the University of New Hampshire. 

4. Megan McAndrew Cooper, editor of the Dartmouth Atlas of Healthcare 
and media relations director for Dartmouth Medical School. 

5. Shirley Dickson, director of reading for the Education Commission of 
the States. 

6. Beverly Showers, consultant to the Iowa Department of Education on pro- 
fessional development. 

7. John Q. Easton, director of research at the Chicago Consortium for 
School Reform. 
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